{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras 概述\n",
    "这个导文讲述的是如何开始 Keras。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入 Keras\n",
    "tf.keras 是tensorflow 对 Keras API 规范的实现，是用于构建和训练模型的高级 API，其中暴扣对 tensorflow 特定功能的支持，比如 eager 执行、tf.data 管道和 Estimators。tf.keras 使 tensorflow 便于使用，并且不会牺牲灵活性和性能\n",
    "\n",
    "tf.keras 能运行任何与 Keras 兼容的代码，但是要记住：\n",
    "1. tensorflow 最新发行版本的 tf.keras 可能与 PyPi 的最新版本 Keras 不同，要检查 tf.keras.version。\n",
    "2. 保存模型的权重时，tf.keras默认为检查点格式。传递 save_format ='h5' 可以使用HDF5（或传递以.h5结尾的文件名）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建一个简单的模型\n",
    "在 Keras，你通过组装图层来创建模型，模型通常是图层，常见的模型类型是一叠图层：tf.keras.Sequential。以下代码是一个全连接网络的组装：\n",
    "\n",
    "可以在[此处](https://tensorflow.google.cn/tutorials/quickstart/beginner)找到有关如何使用顺序模型的完整且简短的示例。\n",
    "\n",
    "要了解有关构建比顺序模型更高级的模型的信息，请参见：\n",
    "- [Guide to the Keras Functional API](https://tensorflow.google.cn/guide/keras/functional)\n",
    "- [Guide to writing layers and models from scratch with subclassing](https://tensorflow.google.cn/guide/keras/custom_layers_and_models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import layers\n",
    "\n",
    "model = tf.keras.Sequential()\n",
    "# 向模型中添加一个包含 64 个单元的密集连接层\n",
    "model.add(layers.Dense(64, activation = 'relu'))\n",
    "# 添加其他\n",
    "model.add(layers.Dense(64, activation = 'relu'))\n",
    "# 添加 10 单元的输出层\n",
    "model.add(layers.Dense(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 配置图层\n",
    "有许多可用的 tf.keras.layers。它们中的大多数共享一些常见的构造函数参数：\n",
    "1. activation：设置图层的激活功能。该参数由内置函数的名称指定或作为可调用对象指定。默认情况下，不应用任何激活。\n",
    "2. kernel_initializer 和 bias_initializer：用于创建层权重（内核和 bias）的初始化方案。此参数是名称或可调用对象。内核默认为 \"Glorot uniform\" 初始化程序，偏差默认为零。\n",
    "3. kernel_regularizer 和 bias_regularizer：应用层权重（内核和 bias ）的正则化方案，例如 L1 或 L2 正则化。默认情况下，不应用任何正则化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x276e05f4048>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建一个 relu layer\n",
    "layers.Dense(64, activation='relu') # or layers.Dense(64, activation=tf.nn.relu)\n",
    "\n",
    "# 将 L1 正则化系数为0.01的线性层应用于内核矩阵：\n",
    "layers.Dense(64, kernel_regularizer=tf.keras.regularizers.l1(0.01))\n",
    "\n",
    "# 将 L1 正则化系数为0.01的线性层应用于偏差向量：\n",
    "layers.Dense(64, bias_regularizer=tf.keras.regularizers.l1(0.01))\n",
    "\n",
    "# 内核初始化为随机正交矩阵的线性层：\n",
    "layers.Dense(64, kernel_initializer='orthogonal')\n",
    "\n",
    "# 偏差向量初始化为2.0的线性层：\n",
    "layers.Dense(64, bias_initializer=tf.keras.initializers.Constant(2.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练和评估\n",
    "模型构建之后，通过调用 compile 方法来定义它的训练过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "    layers.Dense(64, activation='relu', input_shape=(32,)),\n",
    "    layers.Dense(64, activation='relu'),\n",
    "    layers.Dense(10)\n",
    "])\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.Adam(0.01), \n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "             metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[tf.keras.Model.compile](https://tensorflow.google.cn/api_docs/python/tf/keras/Model#compile) 有三个重要的参数：\n",
    "1. optimizer：此对象指定训练的过程。从 [tf.keras.optimizers](https://tensorflow.google.cn/api_docs/python/tf/keras/optimizers) 模块传递优化器实例，例如 [tf.keras.optimizers.Adam](https://tensorflow.google.cn/api_docs/python/tf/keras/optimizers/Adam)、[tf.keras.optimizers.SGD](https://tensorflow.google.cn/api_docs/python/tf/keras/optimizers/SGD)，如果只想使用默认的参数，还可以通过字符串（\"adam\" 或 \"sgd\"）指定优化器。\n",
    "2. loss：起到优化过程中最小化的功能。常见的选择包括均方误差（mse）、categorical_crossentropy 和binary_crossentropy。损失函数可以通过名称指定，也可以通过从 [tf.keras.losses](https://tensorflow.google.cn/api_docs/python/tf/keras/losses) 模块传递可调用对象来指定。\n",
    "3. metrics：用于监视培训。是来自 [tf.keras.metrics](https://tensorflow.google.cn/api_docs/python/tf/keras/metrics) 模块的字符串名称或可调用项。\n",
    "4. 此外，为确保模型能够训练和评估，可以确保将run_eagerly = True作为要编译的参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练来自 numpy 的数据\n",
    "对于小型数据集，可以使用内存中的 numpy 数组来训练和评估模型。该模型使用拟合方法“适合”训练数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples, validate on 100 samples\n",
      "Epoch 1/10\n",
      "1000/1000 [==============================] - 0s 171us/sample - loss: 48020.3911 - accuracy: 0.0900 - val_loss: 42549.6031 - val_accuracy: 0.0900\n",
      "Epoch 2/10\n",
      "1000/1000 [==============================] - 0s 63us/sample - loss: 50606.2599 - accuracy: 0.0800 - val_loss: 71205.2772 - val_accuracy: 0.0800\n",
      "Epoch 3/10\n",
      "1000/1000 [==============================] - 0s 61us/sample - loss: 82337.5261 - accuracy: 0.0750 - val_loss: 89709.3975 - val_accuracy: 0.0800\n",
      "Epoch 4/10\n",
      "1000/1000 [==============================] - 0s 62us/sample - loss: 92648.8911 - accuracy: 0.0970 - val_loss: 121031.4119 - val_accuracy: 0.1400\n",
      "Epoch 5/10\n",
      "1000/1000 [==============================] - 0s 65us/sample - loss: 77264.2364 - accuracy: 0.1030 - val_loss: 53000.0387 - val_accuracy: 0.1200\n",
      "Epoch 6/10\n",
      "1000/1000 [==============================] - 0s 72us/sample - loss: 101339.3819 - accuracy: 0.1130 - val_loss: 147531.4438 - val_accuracy: 0.0700\n",
      "Epoch 7/10\n",
      "1000/1000 [==============================] - 0s 67us/sample - loss: 125185.5177 - accuracy: 0.0980 - val_loss: 151091.5200 - val_accuracy: 0.1400\n",
      "Epoch 8/10\n",
      "1000/1000 [==============================] - 0s 66us/sample - loss: 129444.1996 - accuracy: 0.0870 - val_loss: 103245.9594 - val_accuracy: 0.1100\n",
      "Epoch 9/10\n",
      "1000/1000 [==============================] - 0s 72us/sample - loss: 124400.2918 - accuracy: 0.0960 - val_loss: 178148.1112 - val_accuracy: 0.1300\n",
      "Epoch 10/10\n",
      "1000/1000 [==============================] - 0s 85us/sample - loss: 213931.1803 - accuracy: 0.1020 - val_loss: 176761.3100 - val_accuracy: 0.1400\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2e21708e648>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "data = np.random.random((1000, 32))\n",
    "labels = np.random.random((1000, 10))\n",
    "\n",
    "val_data = np.random.random((100, 32))\n",
    "val_labels = np.random.random((100, 10))\n",
    "\n",
    "model.fit(data, labels, epochs=10, batch_size=32, validation_data=(val_data, val_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[tf.keras.Model.fit](https://tensorflow.google.cn/api_docs/python/tf/keras/Model#fit) 有三个重要参数：\n",
    "1. epochs：构成训练的轮数，一个 epochs 是对整个输入数据的一次迭代。\n",
    "2. batch_size：当传递 numpy 数据时，模型将数据切成较小的批次，并在训练期间对这些批次进行迭代。该整数指定每个批次的大小。如果不能将样本总数除以批次大小，则最后一批为其余数。\n",
    "3. validation_data：在对模型进行原型制作时，传递此参数（输入和标签的元组）可以使模型在每个时期结束时以推断模式显示所传递数据的损失和度量。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用 tf.data 数据集进行训练\n",
    "使用Datasets API可以扩展到大型数据集或多设备培训。将tf.data.Dataset实例传递给fit方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 32 steps\n",
      "Epoch 1/10\n",
      "32/32 [==============================] - 1s 23ms/step - loss: 166692.5682 - accuracy: 0.0930\n",
      "Epoch 2/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 161792.6010 - accuracy: 0.1170\n",
      "Epoch 3/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 197026.9071 - accuracy: 0.0910\n",
      "Epoch 4/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 287651.5585 - accuracy: 0.1080\n",
      "Epoch 5/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 307296.3471 - accuracy: 0.1030\n",
      "Epoch 6/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 344975.0969 - accuracy: 0.1130\n",
      "Epoch 7/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 391247.5488 - accuracy: 0.1020\n",
      "Epoch 8/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 303212.6220 - accuracy: 0.1030\n",
      "Epoch 9/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 414840.5374 - accuracy: 0.0920\n",
      "Epoch 10/10\n",
      "32/32 [==============================] - 0s 2ms/step - loss: 435928.7574 - accuracy: 0.1070\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2e216074d88>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = tf.data.Dataset.from_tensor_slices((data, labels))\n",
    "dataset = dataset.batch(32)\n",
    "\n",
    "val_dataset = tf.data.Dataset.from_tensor_slices((val_data, val_labels))\n",
    "val_dataset = val_dataset.batch(32)\n",
    "\n",
    "# 由于数据集会产生一批数据，因此此代码段不需要batch_size。 数据集也可以用于验证：\n",
    "model.fit(dataset, epochs=10,\n",
    "          validation_data=val_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评估和预测\n",
    "能将 numpy 和 tf.data.Dataset 用于 tf.keras.Model.evaluate 和 tf.keras.Model.predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/1000 [==============================] - 0s 35us/sample - loss: 317876.5768 - accuracy: 0.1190\n",
      "32/32 [==============================] - 0s 3ms/step - loss: 315853.0977 - accuracy: 0.1190\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[315853.09765625, 0.119]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = np.random.random((1000, 32))\n",
    "labels = np.random.random((1000, 10))\n",
    "\n",
    "model.evaluate(data, labels, batch_size=32)\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((data, labels))\n",
    "dataset = dataset.batch(32)\n",
    "\n",
    "model.evaluate(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以下是通过NumPy数组预测对提供的数据进行推断的最后一层的输出的方法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1000, 10)\n"
     ]
    }
   ],
   "source": [
    "result = model.predict(data, batch_size=32)\n",
    "print(result.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建复杂模型\n",
    "### Function API\n",
    "tf.keras.Sequential 模型是不能代表任意模型的简单层堆栈。可以使用 [Keras function API](https://tensorflow.google.cn/guide/keras/functional) 构建复杂的模型拓扑，例如：\n",
    "1. 多输入模型\n",
    "2. 多输出模型\n",
    "3. 具有共享图层（同一图层被调用多次）的模型\n",
    "4. 具有非顺序数据流的模型（例如，残余连接）\n",
    "\n",
    "使用函数式 API 构建模型的工作方式如下：\n",
    "1. 图层实例是可调用的，并返回张量。 \n",
    "2. 输入张量和输出张量用于定义tf.keras.Model实例。\n",
    "3. 该模型像顺序模型一样经过训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用函数式 API 构建简单的全连接网络\n",
    "\n",
    "inputs = tf.keras.Input(shape=(32,))\n",
    "\n",
    "x = layers.Dense(64, activation='relu')(inputs)\n",
    "x = layers.Dense(64, activation='relu')(x)\n",
    "predictions = layers.Dense(10)(x)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=predictions)\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(0.001),\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(data, labels, batch_size=32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 模型子类化\n",
    "通过将tf.keras.Model子类化并定义自己的前向传播，构建完全可定制的模型。在 __init__ 方法中创建图层，并将其设置为类实例的属性。在call方法中定义前向传递。\n",
    "\n",
    "在 eager 模型下，模型子类化特别有用，因为它允许强制性地编写前向传递。\n",
    "\n",
    "> 如果您需要始终强制执行模型，则可以在调用父类构造函数时设置 dynamic = True。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 291us/sample - loss: 12.1971 - accuracy: 0.1130\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 50us/sample - loss: 12.9544 - accuracy: 0.1140\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 47us/sample - loss: 13.7175 - accuracy: 0.1090\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 59us/sample - loss: 14.6415 - accuracy: 0.1150\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 50us/sample - loss: 15.6780 - accuracy: 0.1080\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2e20e7d7988>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyModel(tf.keras.Model):\n",
    "    def __init__(self, num_classes=10):\n",
    "        super(MyModel, self).__init__(name='my_model')\n",
    "        self.num_classes = num_classes\n",
    "        # 定义自己的图层\n",
    "        self.dense_1 = layers.Dense(32, activation='relu')\n",
    "        self.dense_2 = layers.Dense(num_classes)\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        # 定义你的前向传播，使用 __init__ 定义的图层\n",
    "        x = self.dense_1(inputs)\n",
    "        return self.dense_2(x)\n",
    "    \n",
    "# 使用自己定义的模型\n",
    "model = MyModel(num_classes=10)\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(0.001),\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(data, labels, batch_size=32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 自定义图层\n",
    "通过继承 tf.keras.layers.Layer 并重写以下方法创建自定义图层：\n",
    "1. __init__：（可选）定义该层要使用的子层。\n",
    "2. build：创建图层的权重——使用 add_weight 方法添加权重。\n",
    "3. call：定义前向传播。\n",
    "4. Optionally：（可选）可以通过实现 get_config 方法和 from_config 类方法来序列化层。\n",
    "\n",
    "> 要了解更多关于创建图层和模型的信息，可以查看[Guide to writing layers and models from scratch.](https://tensorflow.google.cn/guide/keras/custom_layers_and_models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 1000 samples\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 0s 207us/sample - loss: 11.5124 - accuracy: 0.0890\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 0s 42us/sample - loss: 11.5125 - accuracy: 0.0910\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 0s 39us/sample - loss: 11.5126 - accuracy: 0.0910\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 0s 41us/sample - loss: 11.5124 - accuracy: 0.0880\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 0s 45us/sample - loss: 11.5122 - accuracy: 0.0900\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2e21c520788>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class MyLayer(layers.Layer):\n",
    "\n",
    "  def __init__(self, output_dim, **kwargs):\n",
    "    self.output_dim = output_dim\n",
    "    super(MyLayer, self).__init__(**kwargs)\n",
    "\n",
    "  def build(self, input_shape):\n",
    "    # 为此层创建一个可训练的权重变量。\n",
    "    self.kernel = self.add_weight(name='kernel',\n",
    "                                  shape=(input_shape[1], self.output_dim),\n",
    "                                  initializer='uniform',\n",
    "                                  trainable=True)\n",
    "\n",
    "  def call(self, inputs):\n",
    "    return tf.matmul(inputs, self.kernel)\n",
    "\n",
    "  def get_config(self):\n",
    "    base_config = super(MyLayer, self).get_config()\n",
    "    base_config['output_dim'] = self.output_dim\n",
    "    return base_config\n",
    "\n",
    "  @classmethod\n",
    "  def from_config(cls, config):\n",
    "    return cls(**config)\n",
    "    \n",
    "# 使用自定义图层创建模型\n",
    "model = tf.keras.Sequential([MyLayer(10)])\n",
    "# 编译步骤指定训练配置\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(0.001),\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "# 训练5轮\n",
    "model.fit(data, labels, batch_size=32, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 回调\n",
    "回调是传递给模型的对象，用于在训练期间自定义和扩展其行为。您可以编写自己的自定义回调，也可以使用内置的 [tf.keras.callbacks](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks)，包括：\n",
    "1. [tf.keras.callbacks.ModelCheckpoint](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks/ModelCheckpoint)：定期保存模型的检查点。\n",
    "2. [tf.keras.callbacks.LearningRateScheduler](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks/LearningRateScheduler)：动态改变学习率。\n",
    "3. [tf.keras.callbacks.EarlyStoppint](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks/EarlyStopping)：当验证性能停止改善时中断培训。\n",
    "4. [tf.keras.callbacks.TensorBoard](https://tensorflow.google.cn/api_docs/python/tf/keras/callbacks/TensorBoard)：使用 tensorBoard 可视化模型行为。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [\n",
    "  # 如果`val_loss`停止改善超过2个时期，则进行中断训练\n",
    "  tf.keras.callbacks.EarlyStopping(patience=2, monitor='val_loss'),\n",
    "  # 将TensorBoard日志写入`./ logs`目录\n",
    "  tf.keras.callbacks.TensorBoard(log_dir='./logs')\n",
    "]\n",
    "model.fit(data, labels, batch_size=32, epochs=5, callbacks=callbacks,\n",
    "          validation_data=(val_data, val_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存并还原\n",
    "### 只保存权重值\n",
    "使用 [tf.keras.Model.save_weights](https://tensorflow.google.cn/api_docs/python/tf/keras/Model#save_weights) 保存和加载模型的权重"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "layers.Dense(64, activation='relu', input_shape=(32,)),\n",
    "layers.Dense(10)])\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.Adam(0.001),\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# 将权重保存到TensorFlow Checkpoint文件\n",
    "model.save_weights('./weights/my_model')\n",
    "\n",
    "# 恢复模型状态，这需要具有相同架构的模型。\n",
    "model.load_weights('./weights/my_model')\n",
    "\n",
    "# 默认情况下，这会将模型的权重保存为TensorFlow检查点文件格式。\n",
    "# 权重也可以保存为Keras HDF5格式（Keras的多后端实现的默认设置）：\n",
    "model.save_weights('my_model.h5', save_format='h5')\n",
    "model.load_weights('my_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 仅保存模型配置\n",
    "可以保存模型的配置——这无需任何权重就可以序列化模型体系结构。即使没有定义原始模型的代码，保存的配置也可以重新创建和初始化相同的模型。 Keras 支持 json 和 yaml 序列化格式："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# json 格式保存\n",
    "json_string = model.to_json()\n",
    "import json\n",
    "import pprint\n",
    "pprint.pprint(json.loads(json_string))\n",
    "# 通过 json 新初始化\n",
    "fresh_model = tf.keras.models.model_from_json(json_string)\n",
    "\n",
    "# yaml 格式保存\n",
    "yaml_string = model.to_yaml()\n",
    "print(yaml_string)\n",
    "# 通过 yaml 新初始化\n",
    "fresh_model = tf.keras.models.model_from_yaml(yaml_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 将整个模型保存在一个文件中\n",
    "整个模型可以保存到一个文件中，该文件包含权重值，模型的配置，甚至优化器的配置。这使你可以检查模型并稍后从完全相同的状态恢复训练，而无需访问原始代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "  layers.Dense(10, activation='relu', input_shape=(32,)),\n",
    "  layers.Dense(10)\n",
    "])\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),\n",
    "              metrics=['accuracy'])\n",
    "model.fit(data, labels, batch_size=32, epochs=5)\n",
    "\n",
    "# 将整个模型保存为一个 HDF5 文件\n",
    "model.save('my_model')\n",
    "# 重新创建完全相同的模型，包括权重和优化器。\n",
    "model = tf.keras.models.load_model('my_model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 可从 [save and serialize models](https://tensorflow.google.cn/guide/keras/save_and_serialize) 了解更多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## eager 执行\n",
    "[eager 执行](https://tensorflow.google.cn/guide/eager)是一种立即执行的编程环境，可以立即评估操作。Keras 不需要此功能，但 tf.keras 支持此功能，对检查程序和调试都很有帮助。所有的 tf.keras 那些构建 API 都与 eager 执行兼容。尽管可以使用顺序和函数时 API，但 eager 执行的方法尤其有利于编写前向传递的 API（而不是通过组装现有层来创建模型的 API）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 资源分配\n",
    "### 多 CPU\n",
    "tf.keras 模型可以使用 [tf.distribute.Strategy](https://tensorflow.google.cn/api_docs/python/tf/distribute/Strategy) 在多个 GPU 上运行。该 API 在多个GPU上提供了分布式培训，几乎无需更改现有代码。\n",
    "\n",
    "当前，tf.distribute.MirroredStrategy 是唯一受支持的分发策略。 MirroredStrategy 通过在单个计算机上使用全缩减进行同步训练来进行图形内复制。要使用 distribution.Strategys，要将优化器实例化以及模型构建和编译嵌套在 Strategy 的.scope() 中，然后训练模型。\n",
    "\n",
    "为了获取更多信息，可以查看 [full guide on Distributed Training in TensorFlow](https://tensorflow.google.cn/guide/distributed_training)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.\n",
      "INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)\n",
      "Model: \"sequential_6\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_11 (Dense)             (None, 16)                176       \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 193\n",
      "Trainable params: 193\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Train for 32 steps\n",
      "32/32 [==============================] - 1s 22ms/step - loss: 0.7055\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2e21bfd3f88>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "strategy = tf.distribute.MirroredStrategy()\n",
    "\n",
    "with strategy.scope():\n",
    "  model = tf.keras.Sequential()\n",
    "  model.add(layers.Dense(16, activation='relu', input_shape=(10,)))\n",
    "  model.add(layers.Dense(1))\n",
    "\n",
    "  optimizer = tf.keras.optimizers.SGD(0.2)\n",
    "\n",
    "  model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "                optimizer=optimizer)\n",
    "\n",
    "model.summary()\n",
    "\n",
    "# 接下来，像往常一样在数据上训练模型：\n",
    "x = np.random.random((1024, 10))\n",
    "y = np.random.randint(2, size=(1024, 1))\n",
    "x = tf.cast(x, tf.float32)\n",
    "dataset = tf.data.Dataset.from_tensor_slices((x, y))\n",
    "dataset = dataset.shuffle(buffer_size=1024).batch(32)\n",
    "\n",
    "model.fit(dataset, epochs=1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
